II Sufficient Statistics and Exponential Family

For the parameter estimation problem, we know nothing about the parameter but the observations from such a distribution. Therefore, the observations X1, · · · , Xn is our first hand of information source about the parameter, that is to say, all the available information about the parameter is contained in the observations. However, we know that the estimators we obtained are always functions of the observations, i.e., the estimators are statistics, e.g. sample mean, sample standard deviations, etc. In some sense, this process can be thought of as “compress” the original observation data: initially we have n numbers, but after this “compression”, we only have 1 numbers. This “compression” always makes us lose information about the parameter, can never makes us obtain more information. The best case is that this “compression” result contains the same amount of information as the information contained in the n observations. We call such a statistic as sufficient statistic. From the above intuitive analysis, we can see that sufficient statistic “absorbs” all the available information about θ contained in the sample. This concept was introduced by R. A. Fisher in 1922.